Systems and Methods for Receiving Digital Media and Classifying, Labeling and Searching Offensive Content Within Digital Media

ABSTRACT

Systems and methods for generating and filtering digital media in a multi-player gaming network are disclosed. In a game module stored locally in each client device, a content editor application generates a user interface through which a player may create a personal visual symbol. The player-created personal visual symbol data is processed by a trained classification module and assigned a value indicative of whether the player-created personal visual symbol data is, or is not, permissible in the multi-player gaming network. Based upon the value, an action is applied to the player-created personal visual symbol, including at least one of permitting the player-created personal visual symbol to be used in the multi-player gaming network or prohibiting the player-created personal visual symbol from being used in the multi-player gaming network.

CROSS REFERENCE

The present specification relies on, for priority, U.S. patentProvisional Application No. 62/780,205 entitled “Systems and Methods forReceiving Digital Media and Classifying, Labeling and SearchingOffensive Content Within Digital Media”, filed on Dec. 15, 2018, whichis incorporated by reference herein in its entirety.

FIELD

The present specification is related generally to the field ofmultiplayer online gaming. More specifically, the present specificationis related to systems and methods that receive and process digital mediato classify, label and search player-generated content, particularlyoffensive content, within a gaming environment.

BACKGROUND

Multiplayer online gaming has seen explosive proliferation across theglobe with access to a wide range of age groups. These online gamesallow players with a wide variety of customizable features in order toenhance the overall user experience. One such feature is of enabling theplayers to generate their emblem, badge, banner, coat of arms, mascot,logo or insignia (collectively referred to as a personal visual symbol)as a means of self-expression and motivation during game play. Theplayers are typically allowed to display these personal visual symbolsduring gameplay such as by displaying them on virtual gears, suitsand/or weapons.

Unfortunately, knowingly and sometimes unknowingly, these personalvisual symbols may portray offensive, toxic or objectionable contentsuch as, for example, profane or foul textual content, raciallyinsensitive content, or sexually explicit content. The prior art hasrecognized this problem and attempted to solve it with basic machinelearning models. For example, U.S. Patent Publication No. 2016/0350675discloses a machine learning model trained with features associated withcontent items. Scores are generated based on the model and areassociated with probabilities that the content items includeobjectionable material. U.S. Pat. No. 8,849,911 discloses a contentreview process that generates a confidence score for reported content,where the confidence score comprises a measure of the probability thatthe reported content is inappropriate. Based on the confidence score, asocial networking system either sends a request to the content owner todelete the reported content or sends information to the reporting userabout what actually constitutes inappropriate content and asks them toreconfirm the content report. These approaches, however, are highlyinaccurate and are not tuned to digital media generated by users in avideo game context.

Accordingly, there is still a need for systems and methods thateffectively and efficiently detect and classify player-generatedpersonal visual symbols in the context of a video gaming system ornetwork. There is also a need for systems and methods to search foroffensive or toxic player-generated personal visual symbols that may besimilar, yet a variant, of known offensive symbols, expressions orsentiments. There is further a need for systems and methods to enforce aplurality of content policies and guidelines that prevent use ofobjectionable personal visual symbols or content by the players andinstead ensure use of acceptable expressions within a gamingenvironment.

SUMMARY

The following embodiments and aspects thereof are described andillustrated in conjunction with systems, tools and methods, which aremeant to be exemplary and illustrative, and not limiting in scope. Thepresent application discloses numerous embodiments.

The present specification discloses a method for generating andfiltering digital media in a multi-player gaming network, wherein themulti-player gaming network comprises at least one game server and aplurality of client devices in data communication and located remotefrom each other, the method comprising: executing, in a game modulestored locally in each of the plurality of client devices, a contenteditor application, wherein the content editor application is configuredto generate a user interface through which a player may create apersonal visual symbol and is configured to generate personal visualsymbol data based upon the personal visual symbol; receiving, in the atleast one game server, the player-created personal visual symbol datafrom the game module; processing, in the at least one game server andusing a content classification module, the player-created personalvisual symbol data by submitting the player-created personal visualsymbol data to a trained classification module; assigning, in the atleast one game server and using the content classification module, avalue to the player-created personal visual symbol data wherein thevalue is indicative of whether the player-created personal visual symboldata is or is not permissible in the multi-player gaming network; andapplying an action to the player-created personal visual symbol basedupon said value, wherein the action includes at least one of permittingthe player-created personal visual symbol to be used in the multi-playergaming network or prohibiting the player-created personal visual symbolfrom being used in the multi-player gaming network.

Optionally, the content classification module is configured to augmentthe player-created personal visual symbol prior to processing by thetrained classification module.

Optionally, the personal visual symbol data comprises at least one of animage file or a plurality of rendering instructions in an alphanumericformat.

Optionally, the method further comprises generating multiple personalvisual symbols, in at least one of the plurality of client devices,wherein at least some of the multiple personal visual symbols compriseimagery designed to be not permissible in the multi-player gamingnetwork and at least some of the multiple personal visual symbolscomprise imagery designed to be permissible in the multi-player gamingnetwork. Optionally, the method further comprises receiving the personalvisual symbols which comprise imagery designed to not be permissible inthe multi-player gaming network and the personal visual symbols whichcomprise imagery designed to be permissible in the multi-player gamingnetwork. Optionally, the method further comprises assigning one or morelabels to each of the personal visual symbols which comprise imagerydesigned to not be permissible in the multi-player gaming network andthe personal visual symbols which comprise imagery designed to bepermissible in the multi-player gaming network, wherein each of the oneor more labels comprises a value indicative of whether a personal visualsymbol is or is not to be permitted in the multi-player gaming network.

Optionally, the method further comprises submitting each of the labelledpersonal visual symbols which comprise imagery designed to not bepermissible in the multi-player gaming network and the labelled personalvisual symbols which comprise imagery designed to be permissible in themulti-player gaming network to at least one machine learning module,wherein the at least one machine learning module is configured togenerate the trained classification module. Optionally, at least one ofthe imagery designed to not be permissible in the multi-player gamingnetwork or the imagery designed to be permissible in the multi-playergaming network is submitted to the at least one machine learning modulein a form of alphanumeric text without an accompanying graphical image.

Optionally, the multi-player gaming network automatically applies theaction to the player-created personal visual symbol based upon the valuewithout human intervention.

The present specification also discloses a system for generating andfiltering digital media in a multi-player gaming network, wherein themulti-player gaming network comprises at least one game server and aplurality of client devices in data communication and located remotefrom each other, the system comprising: one or more processors in acomputing device, said one or more processors configured to execute aplurality of executable programmatic instructions to generate and filterdigital media in the multi-player gaming network; a game module storedlocally in each of the plurality of client devices and configured toexecute a content editor application, wherein the content editorapplication is configured to generate a user interface through which aplayer may create a personal visual symbol and is configured to generatepersonal visual symbol data based upon the personal visual symbol; and acontent classification module in the at least one game server,configured to receive and process the player-created personal visualsymbol data by submitting the player-created personal visual symbol datato a trained classification module and to assign a value to theplayer-created personal visual symbol data, wherein the value isindicative of whether the player-created personal visual symbol data isor is not permissible in the multi-player gaming network, and whereinthe content classification module is configured to apply an action tothe player-created personal visual symbol based upon said value, whereinthe action includes at least one of permitting the player-createdpersonal visual symbol to be used in the multi-player gaming network orprohibiting the player-created personal visual symbol from being used inthe multi-player gaming network.

Optionally, the content classification module is configured to augmentthe player-created personal visual symbol prior to processing by thetrained classification module.

Optionally, the personal visual symbol data comprises at least one of animage file or a plurality of rendering instructions in an alphanumericformat.

Optionally, the content editor application is configured to generatemultiple personal visual symbols, in at least one of the plurality ofclient devices, wherein at least some of the multiple personal visualsymbols comprise imagery designed to be not permissible in themulti-player gaming network and at least some of the multiple personalvisual symbols comprise imagery designed to be permissible in themulti-player gaming network. Optionally, the at least one game server isconfigured to receive the personal visual symbols which comprise imagerydesigned to not be permissible in the multi-player gaming network andthe personal visual symbols which comprise imagery designed to bepermissible in the multi-player gaming network. Optionally, the contentclassification module is configured to assign one or more labels to eachof the personal visual symbols which comprise imagery designed to not bepermissible in the multi-player gaming network and the personal visualsymbols which comprise imagery designed to be permissible in themulti-player gaming network, and wherein each of the one or more labelscomprises a value indicative of whether a personal visual symbol is oris not to be permitted in the multi-player gaming network.

Optionally, the content classification module is configured to submiteach of the labelled personal visual symbols which comprise imagerydesigned to not be permissible in the multi-player gaming network andthe labelled personal visual symbols which comprise imagery designed tobe permissible in the multi-player gaming network to at least onemachine learning module, wherein the at least one machine learningmodule is configured to generate the trained classification module.Optionally, at least one of the imagery designed to not be permissiblein the multi-player gaming network or the imagery designed to bepermissible in the multi-player gaming network is submitted to the atleast one machine learning module in a form of alphanumeric text withoutan accompanying graphical image.

Optionally, the content classification module automatically applies theaction to the player-created personal visual symbol based upon the valuewithout human intervention.

The present specification also discloses a computer readablenon-transitory medium comprising a plurality of executable programmaticinstructions wherein, when said plurality of executable programmaticinstructions are executed by a processor in a computing device, aprocess for generating and filtering digital media in a multi-playergaming network is performed, wherein the multi-player gaming networkcomprises at least one game server and a plurality of client devices indata communication and located remote from each other, the plurality ofexecutable programmatic instructions comprising: programmaticinstructions, stored in the computer readable non-transitory medium, forgenerating and filtering digital media in a multi-player gaming networkby: executing, in a game module stored locally in each of the pluralityof client devices, a content editor application, wherein the contenteditor application is configured to generate a user interface throughwhich a player may create a personal visual symbol and is configured togenerate personal visual symbol data based upon the personal visualsymbol; receiving, in the at least one game server, the player-createdpersonal visual symbol data from the game module; processing, in the atleast one game server and using a content classification module, theplayer-created personal visual symbol data by submitting theplayer-created personal visual symbol data to a trained classificationmodule; assigning, in the at least one game server and using the contentclassification module, a value to the player-created personal visualsymbol data wherein the value is indicative of whether theplayer-created personal visual symbol data is or is not permissible inthe multi-player gaming network; and applying an action to theplayer-created personal visual symbol based upon said value, wherein theaction includes at least one of permitting the player-created personalvisual symbol to be used in the multi-player gaming network orprohibiting the player-created personal visual symbol from being used inthe multi-player gaming network.

Optionally, the content classification module is configured to augmentthe player-created personal visual symbol prior to processing by thetrained classification module.

Optionally, the personal visual symbol data comprises a plurality ofrendering instructions in an alphanumeric format representative of animage and does not include an image file.

Optionally, the multi-player gaming network automatically applies theaction to the player-created personal visual symbol based upon the valuewithout human intervention.

The aforementioned and other embodiments of the present shall bedescribed in greater depth in the drawings and detailed descriptionprovided below.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages of the present specificationwill be further appreciated, as they become better understood byreference to the following detailed description when considered inconnection with the accompanying drawings:

FIG. 1A is a block diagram illustrating a multi-player online gamingsystem or environment for implementing offensive content classificationand search workflows, in accordance with embodiments of the presentspecification;

FIG. 1B illustrates a workflow implemented on the system of FIG. 1A forclassifying and labeling offensive content, in accordance with someembodiments of the present specification;

FIG. 2 is a block diagram illustration of a feed-forward machinelearning model configured to perform content classification andlabeling, in accordance with some embodiments of the presentspecification;

FIG. 3 is a flowchart illustrating a plurality of exemplary steps ofimplementing a method of training the machine learning model of FIG. 2,in accordance with some embodiments of the present specification;

FIG. 4 illustrates block diagrams of first and second feed-forwardmachine learning models configured to perform offensive content search,in accordance with some embodiments of the present specification; and

FIG. 5 illustrates a workflow implemented on the system of FIG. 1A forsearching offensive content, in accordance with some embodiments of thepresent specification.

DETAILED DESCRIPTION

In various embodiments, a computing device includes an input/outputcontroller, at least one communications interface and system memory. Thesystem memory includes at least one random access memory (RAM) and atleast one read-only memory (ROM). These elements are in communicationwith a central processing unit (CPU) to enable operation of thecomputing device. In various embodiments, the computing device may be aconventional standalone computer or alternatively, the functions of thecomputing device may be distributed across multiple computer systems andarchitectures.

In some embodiments, execution of a plurality of sequences ofprogrammatic instructions or code enable or cause the CPU of thecomputing device to perform various functions and processes. Inalternate embodiments, hard-wired circuitry may be used in place of, orin combination with, software instructions for implementation of theprocesses of systems and methods described in this application. Thus,the systems and methods described are not limited to any specificcombination of hardware and software.

The term “application programming interface (API)” may refer to a set ofprotocols, routines, functions and/or commands that programmers use todevelop software or facilitate interaction between distinct softwarecomponents or modules.

The term “module” or “component” used in this disclosure may refer tocomputer logic utilized to provide a desired functionality, service oroperation by programming or controlling a general purpose processor.More specifically, a software module or component is a set ofprogrammatic instructions, in the form of routines, functions and/orcommands, and may be referred to as a software package, a web service,or a web resource. It encapsulates a set of related functions (or data)and is separated from another software component by at least one API.All of the data and functions inside each component are semanticallyrelated (just as with the contents of classes). A component is designedto be substitutable, so that a component can replace another component(at design time or run-time), if the successor component meets therequirements of the initial component, as defined by and expressed bythe API(s). Software modules often take the form of objects orcollections of objects from object-oriented programming, in some binaryor textual form, adhering to some interface description language (IDL)so that the module may exist autonomously from other software modules ina computer. Module may be interchangeably used with unit, logic, logicalblock, component, or circuit, for example.

The terms “content” and “personal visual symbol data” are usedinterchangeably throughout the specification.

The term “personal visual symbol” refers to an image, vector or matrixof pixels comprising textual and/or graphical information.

The term “gradient descent” refers to a first-order iterativeoptimization algorithm used in the machine learning models of thepresent specification to find values of parameters (coefficients orweights) of a function (f) that minimizes a cost function (cost). Thus,the gradient descent algorithm works toward adjusting input weights ofthe layers in neural networks and finding local minima or global minimain order to optimize a problem.

The term “stride” refers to the number of pixels a convolution filtershifts over an input matrix of pixels.

The present specification is directed towards multiple embodiments. Thefollowing disclosure is provided in order to enable a person havingordinary skill in the art to practice the invention. Language used inthis specification should not be interpreted as a general disavowal ofany one specific embodiment or used to limit the claims beyond themeaning of the terms used therein. The general principles defined hereinmay be applied to other embodiments and applications without departingfrom the spirit and scope of the invention. Also, the terminology andphraseology used is for the purpose of describing exemplary embodimentsand should not be considered limiting. Thus, the present invention is tobe accorded the widest scope encompassing numerous alternatives,modifications and equivalents consistent with the principles andfeatures disclosed. For purpose of clarity, details relating totechnical material that is known in the technical fields related to theinvention have not been described in detail so as not to unnecessarilyobscure the present invention.

In the description and claims of the application, each of the words“comprise” “include” and “have”, and forms thereof, are not necessarilylimited to members in a list with which the words may be associated. Itshould be noted herein that any feature or component described inassociation with a specific embodiment may be used and implemented withany other embodiment unless clearly indicated otherwise.

As used herein, the indefinite articles “a” and “an” mean “at least one”or “one or more” unless the context clearly dictates otherwise.

FIG. 1A illustrates an embodiment of a multi-player online gaming systemor environment 100 in which offensive content classification/labelingand search workflows of the present specification may be implemented orexecuted. The system 100 comprises client-server architecture, where oneor more game servers 105 are in communication with one or more clientdevices 110 and at least one administrative work-station 145 over anetwork 115. Players may access the system 100 via the one or moreclient devices 110 while at least one administrator may access thesystem 100 using the at least one work-station 145. The client devices110 and the work-station 145 comprise computing devices such as, but notlimited to, personal or desktop computers, laptops, Netbooks, handhelddevices such as smartphones, tablets, and PDAs, gaming consoles and/orany other computing platform known to persons of ordinary skill in theart. Although three client devices 110 are illustrated in FIG. 1, anynumber of client devices 110 can be in communication with the one ormore game servers 105 over the network 115.

The one or more game servers 105 can be any computing device having oneor more processors and one or more computer-readable storage media suchas RAM, hard disk or any other optical or magnetic media. The one ormore game servers 105 include a plurality of modules operating toprovide or implement a plurality of functional, operational orservice-oriented methods of the present specification. In someembodiments, the one or more game servers 105 include or are incommunication with at least one database system 150. The database system150 stores a plurality of game data associated with at least one gamethat is served or provided to the client devices 110 over the network115. In embodiments, the database system 150 also stores a plurality oftraining data. In some embodiments, the one or more game servers 105 maybe implemented by a cloud of computing platforms operating together asgame servers 105.

In accordance with aspects of the present specification, the one or moregame servers 105 provide or implement a plurality of modules such as,but not limited to, a master game module 120, machine learning (ML)modules 125 and 126, a training module 135, a content classificationmodule 130 and a content search module 140. In some embodiments, the oneor more client devices 110 and the administrative work-station 145 areconfigured to implement or execute one or more of a plurality ofclient-side modules that are same as or similar to the modules of theone or more game servers 105. For example, in some embodiments theclient devices 110 execute a client-side game module 120′.

The one or more game servers 105 are preferably configured toconcurrently communicate with at least 20 client devices, and morepreferably 20 to 1,000,000 client devices or any increment therein, suchthat each of said at least 20 client devices are permitted toconcurrently generate, submit, search for, retrieve, and/or index one ormore personal visual symbols. In another embodiment, the one or moregame servers are configured to concurrently host at least 5 requests togenerate, submit, search for, retrieve, and/or index one or morepersonal visual symbols per second, preferably 50-150 requests togenerate, submit, search for, retrieve, and/or index one or morepersonal visual symbols per second, with the plurality of clientdevices.

In some embodiments, the administrative work-station 145 executes atleast one administrative software application that enables theadministrator to interact with the modules 120, 125, 130, 135 and 140using at least one GUI (Graphical User Interface) over the network 115.In some embodiments, the administrator may interact with the modules120, 125, 130, 135 and 140 from the work-station 145 using at least oneweb-based GUI over the network 115. In some embodiments, theadministrative work-station 145 also executes the client-side gamemodule 120′.

It should be appreciated that, in one embodiment, the present inventionachieves at least some of its desired objectives by having the distinctdistribution of modular functionality as shown in FIG. 1A. For example,the administrative work-station executing the administrative softwareapplication is preferably modularly distinct from, and physically remotefrom, one or more of the database 150, machine learning (ML) modules 125and 126, training module 135, content classification module 130 andcontent search module 140. Further, in an embodiment, the trainingmodule 135, content classification module 130 and content search module140 are configured to execute in parallel to each other with each inindependent communication with a ML module 125, 126 and/or the mastergame module 120.

Master Game Module 120

In embodiments, the master game module 120 is configured to execute aninstance of an online game to facilitate interaction of the users withthe game. In embodiments, the instance of the game executed may besynchronous, asynchronous, and/or semi-synchronous. The master gamemodule 120 controls aspects of the game for all players and receives andprocesses each player's input in the game. In other words, the mastergame module 120 hosts the online game for all users, receives game datafrom the client devices 110 and transmits updates to all client devices110 based on the received game data so that the game, on each of theclient devices 110, represents the most updated or current status withreference to interactions of all players with the game. Thus, the mastergame module 120 transmits game data over the network 115 to the clientdevices 110 and the work-station 145 for use by the game module 120′ toprovide local versions and current status of the game to the players andthe administrator, respectively.

Game Module 120′

On the client-side, each of the one or more client devices 110 and theadministrative work-station 145 implements the game module 120′ thatoperates as a gaming application to provide a player with an interfacebetween the player and the game. The game module 120′ generates theinterface to render a virtual environment, virtual space or virtualworld associated with the game and enables the player to interact in thevirtual environment to perform a plurality of game tasks and objectives.The game module 120′ accesses game data received from the game server110 to provide an accurate representation of the game to the player. Thegame module 120′ captures and processes player inputs and interactionswithin the virtual environment and provides updates to the game server110 over the network 115.

In embodiments, the game module 120′ also implements a content editorsoftware application to enable a player to generate virtual personalizedcontent for self-expression such as, for example, an emblem, mascot,symbol, badge, logo or insignia (hereinafter referred to as a “personalvisual symbol”). In various embodiments, the personal visual symbolcomprises textual and/or visual (or graphical) content that the playermay put on their in-game virtual gear. In some embodiments, the contenteditor application is available as a feature within the game module 120′application. A player may launch the content editor from the game module120′ while being in-game or offline. In some embodiments, as aconsequence of the player creating a personal visual symbol, the contenteditor generates the personal visual symbol as an image file as well asa plurality of rendering instructions (similar to postscript or scalablevector graphics) associated with the image file (together referred tohereinafter as “personal visual symbol data”). Thereafter, theplayer-generated personal visual symbol data is uploaded to the server105 for auditing against a plurality of enforceable guidelines orpolicies and manage offensive or toxic content within the system 100.

In accordance with aspects of the present specification, theadministrator may launch the content editor from the game module 120′ onthe work-station 145 to generate offensive as well as inoffensive set ofpersonal visual symbol data for training the ML module 125. In alternateembodiments, the work-station 145 may have a copy of the content editorinstalled as a stand-alone application (independent of the game module120′). It should be appreciated that the presence of the game module120′ on the work-station 145 is to enable the administrator to monitorthat the rendered game is progressing without technical glitches and tointervene for restoring aspects of the game, if needed.

In embodiments, the offensive and inoffensive personal visual symboldata generated by the administrator comprises a first set of trainingdata for training the ML module 125. In embodiments, a sufficientlylarge set of offensive and inoffensive personal visual symbol data isgenerated by the administrator and stored in the database 150.Subsequently, the administrator begins classifying and labeling orranking the first set of training data. For this, in some embodiments,the administrator may access the content classification module 130 fromwork-station 145 through the network 115. On access, the contentclassification module 130 implements a plurality of instructions orprogrammatic code to generate at least one content classification GUI.The GUI is configured to enable the administrator to query the databasefor the first set of training data, present each piece of the first setof training data and allow the administrator to associate one ofoffensive or inoffensive classification to each piece of the first setof training data and also associate a degree of offensiveness label,ranking or score (on a predetermined scale of offensiveness such as, ofexample, a numerical scale of 1 to 5 where the degree of offensivenessincreases from 1 to 5) with the data classified as offensive.

The human-labeled first set of training data is stored in the database150 for retrieval for the purposes of training the ML module 125.

Machine Learning (ML) Module 125

In accordance with an aspect of the present specification, the ML module125 executes a plurality of instructions or programmatic code toimplement a machine learning model that receives personal visual symboldata as input, processes the personal visual symbol data and outputs aclassification and label corresponding to the personal visual symboldata.

In various embodiments, the machine learning model may include one ormore support vector machines, linear regression models, clusteringanalysis models, boosted decision trees, neural networks, deep learningmodels or a combination thereof. In some embodiments, the machinelearning model is a deep learning feed-forward network such as amultilayer convolutional neural network (CNN).

FIG. 2 is a block diagram illustration of a feed-forward machinelearning model 200 configured to perform content classification andlabeling, in accordance with some embodiments of the presentspecification. In one embodiment, the model 200 is a multilayer CNN inwhich each convolutional layer 205 is connected to every other layer ina feed-forward fashion. In some embodiments, for each layer 205, featuremaps of all preceding layers are used as inputs, and its ownfeature-maps are used as inputs into all subsequent layers 205. Thus, topreserve the feed-forward nature, each layer 205 obtains additionalinputs from all preceding layers 205 and passes on its own feature-mapsto all subsequent layers 205. In various embodiments, the model 200comprises a block 210 of ‘n’ convolutional layers 205 where ‘n’ isgreater than or equal to 3. In some embodiments, the block 210 includes‘n’=12 convolutional layers 205. For processing, content data 215(offensive and inoffensive) is provided to the model 200 as input and apredicted offensiveness score and classification are received as outputat a final classification layer 220. It should be appreciated that, invarious embodiments, the input content 215 comprises textual and/orvisual (graphical or image) data. In some embodiments, the content 215is personal visual symbol data.

In embodiments, to facilitate down-sampling, the model 200 is furtheradapted to include a plurality of blocks 210 separated by poolingtransition layers. In some embodiments, for image inputs of 224×224pixels (for example, resized from 256×256 bitmap images), a 7×7 initialconvolutional down-sampling (stride=2) is used followed by a 3×3 maxpooling (stride=2) followed by three blocks 210 of 12 convolutionallayers each, and finally followed by a global average pooling acrosschannels. In embodiments, an output of the global average pooling is avector of length ‘m’. For example, in an embodiment, where a layer shape(that is, matrix shape) prior to the global average pooling is14×14×456, the response/output of the global average pooling is a vectorof length m=456.

Persons of ordinary skill in the art would understand that each layer205 of the block 210 has a weight matrix 212 associated therewith thatis determined during learning, also referred to as a training stage.

Referring back to FIG. 1A, in the training stage, at least one set oftraining data (for example, a training set of personal visual symboldata each having a known output) is processed by the ML module 125 tolearn how to provide an output for new player-generated input data bygeneralizing the information the ML module 125 learns in the trainingstage from the training data. The weight matrices can be adjusted andattuned further based on experience, making the ML module 125 adaptiveto inputs and capable of learning.

Training Module 135

Referring now to FIG. 1A, in some embodiments, a training module 135implements a plurality of instructions or programmatic code to manageand control initial training of the ML module 125. In variousembodiments, the training module 135 accesses at least one set oftraining data from the database system 150, provides the training data(in accordance with a training schedule, for example) as input to the MLmodule 125 for processing using at least learning algorithm and may alsomonitor (such as, for example, in case of supervised training) theoutput generated by the ML module 125.

In embodiments, the database system 150 has stored the first set oftraining data comprising administrator generated, classified and labeledpersonal visual symbol data for supervised training. In someembodiments, the database system 150 also stores a second set oftraining data for unsupervised training. The second set of training datais characterized by the fact that the data is not classified and/orlabeled as offensive or inoffensive. In some embodiments, the second setof training data may comprise unlabeled or unclassified player generatedpersonal visual symbol data existing in the database 150 prior toimplementing the toxicity detection methods of the presentspecification. In some embodiments, the database system 150 alsooptionally stores a third set of training data for supervised training.In some embodiments, the first, second and third sets of training dataare stored in separate schemas of the database system 150. In someembodiments, the training module 135 implements a training schedulewherein the ML module 125 is trained using the first set of trainingdata (for supervised training). In some embodiments, the training module135 implements a training schedule wherein the ML module 125 is trainedusing the second set of training data (for unsupervised training)followed by the first set of training data (for supervised training). Insome embodiments, the training module 135 implements a training schedulewherein the ML module 125 is trained using the second set of trainingdata (for unsupervised training), followed (optionally) by the third setof training data (for supervised training) and finally using the firstset of training data (for supervised training).

In embodiments, the first set of training data comprises a plurality ofhuman-labeled and classified personal visual symbol data having textualand/or image (or graphical) content. In other words, each piece of thepersonal visual symbol data has a known output—that is, is alreadyclassified as offensive or inoffensive, wherein the offensive content isalso labeled with a degree of offensiveness.

In embodiments, the training module 135 presents the first set oftraining data to the ML module 125 for processing, as part of supervisedtraining. Supervised training comprises enabling the ML module 125 tolearn a function that maps one or more inputs (first set of trainingdata) to one or more known outputs (human labeled and classified). Sincethe outputs for each of the first set of training data is already known,a learning algorithm of the ML module 125, for supervised learning,iteratively makes predictions on the first set of training data and iscorrected by a feedback from the training module 135 when thepredictions are off with respect to the known outputs.

The learning algorithm analyzes the first set of training data andproduces an inferred function, which can be used for mapping newcontent. An optimal scenario allows for the algorithm to correctlydetermine the classification and labels for unseen or new content. Thisrequires the learning algorithm to generalize from the first set oftraining data to unseen situations. In various embodiments, the learningalgorithm is a gradient descent algorithm. In some embodiments, thelearning algorithm is a stochastic gradient descent. In someembodiments, the learning algorithm is a batch gradient descent. In someembodiments, the learning algorithm is a mini-batch gradient descent.The goal of the gradient descent algorithm is to find parameters (forexample, coefficients or weights) that minimize an error of the ML model125 on the first set of training dataset. The algorithm does this bymaking changes to the parameters that move it along a gradient or slopeof errors down toward a minimum error value.

As described earlier, the first set of training data, human-labeled andclassified, comprises personal visual symbol data—that is, personalvisual symbol image files and rendering instructions associated witheach of the image files. Thus, a personal visual symbol image in thefirst set of training data is also represented by rendering instructions(together with or instead of a vector/matrix of pixels) such as, forexample, “place Symbol 1 at location (100, 100) with scale 1.0 androtation 0.25; place Symbol 2 at location (100, 100) with scale 1.0 androtation 0.75”. The following is an exemplary set of renderinginstructions, representing a personal visual symbol image, with aplurality of layers providing instructions (similar to postscript orscalable vector graphics) on how the personal visual symbol image shouldbe rendered in-game:

“personal visual symbol”: { “image”: { “layers”: [ { “iconID”: 294,“materialID”: 255, “colorLinearSrgb”: { “r”: 46, “g”: 25, “b”: 0, “a”:255 }, “color1LinearSrgb”: { “r”: 46, “g”: 25, “b”: 0, “a”: 255 },“pos_x”: 127, “pos_y”: 127, “scale_x”: 117, “scale_y”: 127, “angle”: 0,“materialPos_x”: 127, “materialPos_y”: 127, “materialScale_x”: 127,“materialScale_y”: 127, “materialAngle”: 0, “gradientAngle”: 0,“gradientOffset”: 127, “gradientSpread”: 127, “mode”: 8, “outline”:false, “flip”: false, “blend”: false, “linearGradient”: false,“editorType”: 4 }, ... ] } }

In accordance with some aspects of the present specification, instead ofrendering the instructions data to a vector of pixels (such as, forexample, a PNG or JPEG file) and presenting the vector of pixels to theML module 125 for training/learning, the rendering instructions for eachimage (in the first set of training data) is directly fed as input tothe ML module 125 for training. This enables the ML module 125 to learnthat, for example, a set of instructions, having a plurality of textstrings in a certain configuration, represents a negative racial symbolsuch as a swastika. Using the rendering instructions as input fortraining has a benefit of circumventing the need to render theinstructions to a pixel array. This would enable having direct access totextual data (either standalone or in combination with graphical data)without requiring to OCR (Optical Character Recognition) and asimplified data representation since there are 256{circumflex over( )}(256*256*3) unique pixel vectors that can be created whereas thereare probably far fewer unique representations using renderinginstructions.

In some embodiments, rendering instructions in the first set of trainingdata are fed directly to the ML module 125 for training. In someembodiments, both personal visual symbol image (vector of pixels) andrendering instructions in the first set of training data are feddirectly to the ML module 125 for training. In some embodiments,personal visual symbol images (vector of pixels) in the first set oftraining data are fed to the ML module 125 for training and generationof learning features/feature vectors that enable the ML module 125 torecognize textual words as a function of the training task. In someembodiments, textual data is extracted (such as by using opticalcharacter recognition (OCR)) and is fed in combination with pixel datato the ML module 125 for training.

In some embodiments, the training module 135 augments supervisedtraining by accessing the third set of training data from the database150 and presenting to the ML module 125 for processing. The third set oftraining data comprises one or more classified and/or labeled publiclyavailable open datasets (of image and textual content) such as, but notlimited to, MNIST, MS-COCO, ImageNet, Open Images, VisualQA, CIFAR-10,CIFAR-100, Sentiment Labeled Sentences Dataset, and SNLI Corpus.

After supervised training, in some embodiments, the training module 135accesses the second set of training data from the database 150 andpresents to the ML module 125 for processing, as part of unsupervisedtraining. Unsupervised training enables the ML module 125 to learn fromthe second set of training data that has not been labeled, classified orcategorized. Instead of responding to feedback from the training module135, unsupervised learning identifies commonalities in the training dataand reacts based on the presence or absence of such commonalities ineach piece of training data. A learning algorithm for unsupervisedlearning is left to itself to discover and present the underlyingstructure in the training data. In some embodiments, the learningalgorithm of the ML module 125, for supervised learning, is gradientdescent based (such as, stochastic, batch and mini-batch) with amodified cost function that includes a term such as, but not limited to,an input reconstruction term, a term based on the joint distributionbetween inputs and learned variables, or an adversarial term. In someembodiments, the learning algorithm of the ML module 125, forunsupervised learning, includes Hebbian learning.

FIG. 3 is a flowchart illustrating a plurality of exemplary steps ofimplementing a method of training the ML module 125, in accordance withsome embodiments of the present specification. Referring now to FIGS. 1Aand 3, at step 305, the first set of training data is accessed by thetraining module 135, from the database system 150 for supervisedtraining of the ML module 125. At step 310, the training module 135provides sample personal visual symbol data of the first set of trainingdata as input to the ML module 125 that in some embodiments is aconvolutional neural network (CNN) in which each convolutional layer isconnected to every other layer in a feed-forward fashion. In someembodiments, the personal visual symbol data comprises personal visualsymbol image file or vector of pixels and associated renderinginstructions. In some embodiments, the personal visual symbol dataincludes only the rendering instructions associated with a personalvisual symbol image file or vector of pixels.

At step 315, as a result of the input personal visual symbol data, theML module 125 performs forward propagation to generate at least oneoutput comprising offensive/inoffensive classification and a label orranking of the degree of offensiveness in case of an offensiveclassification. At step 320, the training module 135 determines an errorbetween the generated output and the known output of the sample personalvisual symbol data (since the personal visual symbol data ishuman-labeled for supervised training).

If the output is incorrect then, at step 325, in accordance with alearning algorithm—back propagation is performed according to thedifference between the generated output and the known output to correctparameters (such as, for example, the coefficients or weight matrices)of the ML module 125. If the output is correct, then the flow moves backto step 310 to continue inputting personal visual symbol data to the MLmodule 125 for processing.

In some embodiments, the learning algorithm is stochastic gradientdescent that calculates the error and updates the parameters of the MLmodule 125 for each sample in the first set of training data. In someembodiments, the learning algorithm is batch gradient descent thatcalculates the error for each sample in the first set of training data,but only updates the parameters of the ML module 125 after all trainingexamples have been evaluated. In some embodiments, the learningalgorithm is mini-batch gradient descent that splits the first set oftraining data into small batches that are used to calculate the errorand update the ML module 125 parameters.

At step 330, the training module 135 determines if the ML module 125 hasgone through a predefined maximum number of training iterations orpasses using the first set of training data. If the predefined maximumnumber of training iterations are met then, at step 335, the trainingends else the flow control moves back to step 310. In some embodiments,the training module 135 may additionally determine if an error rate ofthe ML module 125, on the first set of training data, reaches or islower than a predetermined value. If the predetermined error rate is metprior to the ML module 125 completing the predefined maximum number oftraining iterations then the training module 135 may employ “earlystopping” of the training at step 335.

At step 330, the training module 135 determines if an error rate of theML module 125, on the first set of training data, reaches or is lowerthan a predetermined value. If the predetermined value is met then, atstep 335 the training ends else the flow control moves back to step 310.

In some embodiments, the training module 135 augments the supervisedtraining steps 310 to 335 by accessing the third set of training datafrom the database 150 and presenting to the ML module 125 forprocessing. As discussed earlier in the specification, the third set oftraining data comprises one or more classified and/or labeled publiclyavailable open datasets (of image and textual content).

In some embodiments, prior to supervised training (using the first setof training data followed by the third set of training data), thetraining module 135 accesses the second set of training data from thedatabase 150 and presents to the ML module 125 for processing, as partof unsupervised training.

In embodiment, the training process results in a trained ML module 125′that includes various processing layers, each with a learnt weightmatrix. The trained ML module 125′ takes as input a representation ofplayer-generated personal visual symbol data (for example, a matrix orvector of pixels and/or associated rendering instructions) and passesthe representation through a plurality of transforms such as, but notlimited to, edge detection, shape detection, and compression. Eachtransformation enables the trained ML module 125′ to better understandwhat the personal visual symbol data represents/contains and ultimatelyclassify the personal visual symbol data and predict its offensiveness.

In some embodiments, once training is complete, a validation dataset isprocessed by the trained ML module 125′ to validate the results oftraining/learning. Finally, player-generated personal visual symbol data(for which generating an output is desired) can be processed by avalidated and trained ML module 125′ and the results stored in thedatabase system 150.

Content Classification Module 130

In embodiments, the player-generated personal visual symbol data(uploaded to the server 105 and stored in the database system 150) isaccessed or queried by the content classification module 130 to initiateprocessing by the trained ML module 125′. In accordance with aspects ofthe present specification, the content classification module 130implements a plurality of instructions or programmatic code to manageprocessing of the player-generated personal visual symbol data, fordetecting offensive content, using the trained machine learning (ML)module 125. In some embodiments, the classification module 130 providesthe player-generated personal visual symbol data as input to the trainedML module 125′, also referred to as a trained classification module,that processes the player-generated personal visual symbol data andoutputs a classification and label corresponding to the personal visualsymbol data. In some embodiments, only the personal visual symbol imagefile is provided as input to the trained ML module 125′. In someembodiments, only the rendering instructions (associated with thepersonal visual symbol image file) are provided as input to the trainedML module 125′. In some embodiments, where only the personal visualsymbol image file is input to the trained ML module 125′, the personalvisual symbol image file is subjected to a plurality of pre-processingfunctions (for image data augmentation) such as, but not limited to,shifting, zooming, rotating by up to, for example, 20% with randomhorizontal flips.

In some embodiments, the content classification module 130 assigns avalue to the player-generated personal visual symbol data based on theclassification and label output by the trained classification module,wherein the value is indicative of whether the player-created personalvisual symbol data is or is not permissible. An action is then appliedto the player-created personal visual symbol based on the value. In someembodiments, the action includes at least one of permitting theplayer-created personal visual symbol to be used in a multi-playergaming network or prohibiting the player-created personal visual symbolfrom being used in a multi-player gaming network. In some embodiments,the multi-player gaming network automatically applies the action to theplayer-created personal visual symbol based upon the value without humanintervention.

In embodiments, the classification parameter predicts whether thepersonal visual symbol is offensive or not while the label parameterpredicts a degree of offensiveness or toxicity of the personal visualsymbol. In some embodiments, the degree of offensiveness is embodied asa score, for example, on a predefined scale. For example, the predefinedscale may be a 1 to 5 numerical scale where the degree of offensivenessincreases from 1 to 5 (1 being the lowest and 5 being the highest degreeof offensiveness). In some embodiments, the degree of offensiveness maydetermine whether the player (who generated the offensive personalvisual symbol) is permanently or temporarily banned from participatingin the gaming system 100 in accordance with content enforcement policiesand guidelines. In some embodiments, a player may be the originalcreator of an offensive personal visual symbol and may share the symbolwith one or more other players. In such circumstances, the player who isthe original creator of the symbol may be permanently banned while theone or more other players may be subject to a temporary ban. A permanentban means that the video game is configured to prevent or stop theplayer from engaging in gameplay by 1) blocking a hardware addressassociated with the player, 2) blocking a network address associatedwith the player, 3) deleting or deactivating an account associated withthe player, 4) prohibiting the player from re-entering the game based onhis or her user identification, or 5) prohibiting the player fromrejoining the game under a different user identification if the playername, network address, and/or hardware address is the same as the bannedplayer's corresponding data. A temporary ban is technically similar tothe permanent ban except subject to a predefined time period, such asone day, one week, one month, or one year or any time increment therein.

The following are exemplary offensiveness criteria and associated typeof enforcement:

-   -   Degree of offensiveness score of 1 to 3 (on the scale of 1 to        5)—minor enforcement leading to temporary ban of the player. The        score of 1 to 3 being indicative of offensive imagery        containing, for example, sexually explicit images and/or foul        language;    -   Degree of offensiveness score of 4 or 5 (on the scale of 1 to        5)—major enforcement leading to permanent ban of the player. The        score of 4 or 5 being indicative of content depicting racism,        bigotry, and/or discriminatory or hateful imagery, for example.

In some embodiments, the content classification module 130 enablesautomatic enforcement (permitting or prohibiting the personal visualsymbol in the multi-player gaming network) as a consequence of theclassification and labeling output by the trained ML module 125′ withoutfurther human intervention. In some embodiments, the contentclassification module 130 enables the administrator to audit and verifythe classification and labeling output by the trained ML module 125′. Insome embodiments, the content classification module 130 enablessupervised enforcement as a consequence of the classification andlabeling output by the trained ML module 125′.

In some embodiments, the content classification module 130 implements aplurality of instructions or programmatic code to generate at least oneverification GUI. In some embodiments, the verification GUI isaccessible to the administrator from his work-station 145 through thenetwork 115. In embodiments, the verification GUI enables theadministrator to query the database system 150 for player-generatedpersonal visual symbol data processed by the ML module 125 during aspecified period of time (for example, the administrator may query forpersonal visual symbol data generated by all players and processed bythe ML module 125 during the last one week), enables the queriedplayer-generated personal visual symbol data to be presented to theadministrator along with the associated classification and labeling as aresult of processing by the ML module 125, enables the administrator toaudit and verify if the classification and labeling is accurate for eachof the player-generated personal visual symbol data, enables theadministrator to attach his verification feedback to the classificationand labeling for each of the player-generated personal visual symboldata wherein the verification feedback is indicative of whether theclassification and labeling is correct or erroneous along with a correctclassification and labeling in case of erroneous processing by the MLmodule 125, and enables saving the administrator audited and verifiedpersonal visual symbol data to the database system 150.

In some embodiments, the queried player-generated personal visual symboldata is presented to the administrator (along with the associatedclassification and labeling) using active learning techniques (such as,for example, uncertainty sampling) for administrator training andperformance evaluation.

In accordance with aspects of the present specification, once theadministrator-audited and verified personal visual symbol data is savedto the database system 150, the content classification module 130 issuesan event flag to the training module 135. As a result, the trainingmodule 135 queries the database system 150 for administrator verifiedand classified and labeled personal visual symbol data and feeds thedata to the ML module 125 for continuous supervised training/learningand improvement of the ML module 125.

In some embodiments, the content classification module 130 implements aplurality of instructions or programmatic code to generate at least oneenforcement GUI. In some embodiments, the enforcement GUI enables theadministrator to query the database system 150 for administrator-auditedand verified personal visual symbol data during a specified period oftime and having a specified associated classification and labeling. Forexample, the administrator may use the enforcement GUI to query andconsequently view all player-generated personal visual symbol data thathave been audited and verified by the administrator over a period oftime, e.g. the last one day, week, or month, and that have been verifiedby the administrator to be offensive. Depending upon the labeling orranking indicative of the degree of offensiveness, the administrator mayattach temporary or permanent enforcement tags to the correspondingpersonal visual symbol data. Thereafter, the enforcement tags are savedto the database system 150. In some embodiments, once the enforcementtags are saved to the database system 150, the content classificationmodule 130 issues an event flag to the master gaming module 120 thatexecutes a plurality of programmatic instructions to implement theenforcements within the system 100. In some embodiments, the contentclassification module 130 may itself be configured to implement theenforcements within the system 100.

Additionally, in some embodiments, once the enforcement tags are savedto the database system 150, the content classification module 130 alsoissues an event flag to the training module 135. As a result, thetraining module 135 queries the database system 150 for enforcementtagged personal visual symbol data and feeds the data to the ML module125 for continuous supervised training/learning and improvement of theML module 125.

In various embodiments, the player-generated personal visual symbol datapresented to the administrator, via verification and/or enforcementGUIs, may be biased. For example, the ML module 125 may be adept atdetecting certain personal visual symbols, e.g., swastikas, and thustend to identify and present swastikas to the administrator forpotential enforcement. The administrator reviewing the results wouldconfirm the swastikas are offensive (in accordance to content policiesand guidelines) thereby reinforcing the existing learning of the MLmodule 125 to continue to identify and present swastikas. This may leadto training the ML module 125 to do something that is already good at,as opposed to becoming more adept at identifying other offensiveimagery.

In embodiments, in order to mitigate this biasing problem, the trainingmodule 135 and/or the content classification module 130 implements aplurality of programmatic instructions or code to a) inject (based onsome heuristic) textual and/or image (graphical) content not predictedto be offensive into the results exposed or presented to theadministrator via the verification and/or enforcement GUIs (for example,for every 1000 predicted offensive images presented for review, 50random images are included as well) and/or b) modify the ML model 125 topenalize personal visual symbol data that the module 125 is alreadyconfident in so that a more diverse dataset is presented to theadministrator.

In various other embodiments, the biasing problem may be mitigated usingmethods such as, but not limited to, sample set bias correction using anauxiliary model, hard example mining, and/or incorporating unsupervisedmetrics into the cost function.

Sample set bias correction using an auxiliary model: This approach tocorrecting sampling bias is directed towards recovering the datadistributions of the training and validation data and then performingcorrections based on the distribution estimates. In some embodiments ofthe present specification it is desirable to recover the datadistribution of biased labeled data and the data distribution of anunbiased sample of all data for correction. It should be appreciatedthat this approach works for low-dimensional feature spaces andtypically the CNN model of the present specification reduces thedimensionality of data. However, it is desirable for the CNN model ofthe present specification to be unbiased. A solution is to use, in someembodiments, an auxiliary CNN model trained on an unbiased dataset(either publicly available, or trained in an unsupervised manner on anunbiased sample of data).

Hard example mining: This approach of dealing with data imbalance isdirected towards weighing the cost of examples proportional to theirrepresentation in the data. This works when the data classes are known,but in dealing with “within-class imbalance” (that is, bias), it isrequired to determine which examples are overrepresented. A solution isto use hard example mining, in some embodiments, which uses the CNNmodel's cost function to determine the “difficulty” of each example,which can then be used to adjust the effective cost through repetitionof hard examples or omission of easy examples.

FIG. 1B illustrates a workflow 160 implemented on the system 100 of FIG.1A for detecting, classifying and labeling offensive content, inaccordance with some embodiments of the present specification. Referringnow to FIGS. 1A and 1B, at step 162 the training module 135 queries thedatabase system 150 to access a plurality of training datasets such asthe first, second and third sets of training data. In embodiments, thefirst and third set of training data are human-labeled for supervisedtraining while the second set of training data is unlabeled andunclassified for unsupervised training. At step 164, the training module135 implements supervised and unsupervised training of the ML module 125using the first, second and third sets of training data. For supervisedtraining, the first and third sets of training data comprise a pluralityof human-labeled offensive content 163 a and inoffensive content 163 b.In some embodiments, the content 163 a, 163 b comprises personal visualsymbol data. Output of the supervised and unsupervised training is thetrained ML module 125′ that, at step 166, is deployed for use within thesystem 100.

At step 168, the content classification module 130 presents a pluralityof player-generated personal visual symbol data 169 to the trained MLmodule 125′ for classification and labeling in terms of beingoffensive/inoffensive and a ranking or score indicative of a degree ofoffensiveness. At step 170, the trained ML module 125′ processes theplayer-generated personal visual symbol data 169 and predictsoffensive/inoffensive classification along with a degree ofoffensiveness as output. At step 172, the output of the trained MLmodule 125′ along with the corresponding player-generated personalvisual symbol data 169 is saved in the database system 150. In someembodiments, the plurality of player-generated personal visual symboldata 169 may first be saved to the database system 150 and laterpresented to the trained ML module 125′ for processing and the resultingoutput is again saved to the database system 150.

At step 174, at least one administrator queries the database system 150for verification of the classification and labeling (of theplayer-generated personal visual symbol data 169) by the trained MLmodule 125′. The queried personal visual symbol data is presented to theadministrator in at least one verification GUI 176. The verification GUI176 enables the administrator to audit and verify if the classificationand labeling is accurate for each of the player-generated personalvisual symbol data, enables the administrator to attach his verificationfeedback to the classification and labeling for each of theplayer-generated personal visual symbol data wherein the verificationfeedback is indicative of whether the classification and labeling iscorrect or erroneous along with a correct classification and labeling incase of erroneous processing by the trained ML module 125′. At step 178the administrator-audited and verified personal visual symbol data issaved to the database system 150. In embodiments, the administratoraudited and verified personal visual symbol data is also available forquerying at step 162 for the purposes of supervised training.

At step 180, at least one administrator queries the database system 150for enforcement of predefined policies and guidelines with respect tooffensive/inoffensive player-generated personal visual symbol dataclassified and labeled by the trained ML module 125′. In someembodiments, enforcement is implemented using the player-generatedpersonal visual symbol data that has also been audited and verified bythe administrator (at step 174). The queried personal visual symbol datais presented to the administrator in at least one enforcement GUI 182.The enforcement GUI 182 enables the administrator to attach or associatetemporary or permanent enforcement tags to the corresponding personalvisual symbol data depending upon whether the personal visual symboldata is classified as offensive and based on the degree of offensivenessof the personal visual symbol data. Thereafter, the enforcement tags aresaved to the database system 150 for subsequent enforcement and forsupervised training at step 162.

Personal Visual Symbol Search Function

FIG. 4 illustrates block diagrams of first and second feed-forwardmachine learning models configured to perform content search or reversesearch, in accordance with some embodiments of the presentspecification. Referring to FIGS. 1A and 4, in accordance with an aspectof the present specification, the ML module 126 executes a plurality ofinstructions or programmatic code to implement first and second machinelearning models 400 a, 400 b that respectively receive 2D (twodimensional) texture data 410 and 3D (three dimensional) model data 411as input data, process the input data through a plurality ofconvolutional layers 415 and output “signatures” 420, 421 representativeof the input data and predicted images 425, 426 similar to the“signatures” 420, 421, wherein the predicted images 425, 426 (that themodels 400 a, 400 b find similar to the inputs 410, 411) are queried andaccessed by the models 400 a, 400 b from the database system 150.Alternatively, in some embodiments, personal visual symbolclassification/labeling as well as search are performed by the samemodel—that is, by the model 200 of FIG. 2. In such embodiments, a costfunction of the model 200 of FIG. 2 is modified to incorporate a visualsimilarity term to enable the model 200 to perform the searchfunctionality.

In some embodiments, for each layer 415, feature maps of all precedinglayers are used as inputs, and its own feature-maps are used as inputsinto all subsequent layers 415. Thus, to preserve the feed-forwardnature, each layer 415 obtains additional inputs from all precedinglayers 415 and passes on its own feature-maps to all subsequent layers415. In various embodiments, the models 400 a, 400 b respectivelycomprise blocks 430, 431 of ‘n’ convolutional layers 415 where ‘n’ ispreferably equal to or greater than 3. In some embodiments, ‘n’=12. Inembodiments, the models 400 a, 400 b are further adapted to respectivelyinclude a plurality of blocks 430, 431 separated by pooling transitionlayers. Persons of ordinary skill in the art would understand that eachlayer 415 of the blocks 430, 431 has a weight matrix 435 associatedtherewith that is determined during learning, also referred to as atraining stage.

In some embodiments, the 2D texture data comprise a matrix or vector ofpixel values. Accordingly, assets in the form of 2D texture data arestored in the form of datasets comprising a plurality of matrices orvectors, each comprising pixel values. In some embodiments, the 3D modeldata comprise point cloud or mesh representation of 3D image contentwhere the mesh representation comprises a collection of vertices, edgesand faces that define the shape of a polyhedral object. The facesusually consist of triangles (triangle mesh), quadrilaterals, or othersimple convex polygons, since this simplifies rendering, but may also becomposed of more general shapes, concave polygons, or polygons withholes. Accordingly, assets in the form of 3D image content are stored inthe form of datasets comprising a plurality of related or connectedvertices, edges and faces that define the shape of a polyhedral object,with points therein comprising pixel values. In embodiments, the term“signature” refers to a vector or matrix of numbers of length ‘m’. Forexample, in the first and second machine learning models 400 a, 400 b,“structures” 420, 421 refer to vectors of numbers of length ‘m’=456,representing global average pooled responses of each input data 410,411. A signature, also referred to as a feature vector, is a visualcharacteristic or element, short of an entire image, that is indicativeof certain types of assets, such as offensive, copyrighted, or otherwiseprohibited content.

In embodiments, training of the models 400 a, 400 b is managed by thetraining module 135. In embodiments, the training module 135 queries thedatabase system 150 that stores a plurality of indexed 2D texture and 3Dmodel data 410, 411. The plurality of indexed 2D texture and 3D modeldata 410, 411 is fed as input to the models 400 a, 400 b to generateoutput signatures that are used by the models 400 a, 400 b to query(from the database system 150) and predict images 425, 426 similar tothe input. In some embodiments, the models 400 a, 400 b query and searchimages 425, 426 (similar to the input data 410, 411) using a metric suchas L2 (Euclidean) distance or cosine angle. For example, for an L2 indexcomprising A:[0,0,1], B:[1,0,1] and C:[0,1,1] a query of [0,0,9] wouldreturn A as most similar. Thereafter, the training module 135 determineswhether the predicted images are correct or erroneous. If erroneous, themodels 400 a, 400 b re-configure their parameters, such as coefficientsand weights 435, using a gradient descent algorithm such as stochastic,batch or mini-batch.

Content Search Module 140

In embodiments, the content search module 140 manages reverse searchfunction using the first and second machine learning models 400 a, 400 bthat have been trained. To initiate reverse search—that is, to search 2Dand/or 3D personal visual symbol data similar to input 2D and/or 3Dcontent—the content search module 140 provides 2D/3D content as input tothe models 400 a, 400 b, obtains the signatures 420, 421 output by themodels 400 a, 400 b and stores the signatures 420, 421 in relation tothe input content in the database system 150. The content search module140 also directs the models 400 a, 400 b to query the database system150 to search player-generated personal visual symbol data similar tothe input content and present the queried output on at least one GUI.

In accordance with aspects of the present specification, a datastructure is used to store n² pairwise similarities where n is of theorder of tens of millions. In embodiments, the data structure includesstructures such as, but not limited to, k-d tree (a binary search treewhere data in each node is a k-dimensional point in space) and learnedhash map. In some embodiments, the system of the present specificationuses learned quantization of ‘m’ dimension vectors to ‘n’ dimensions andstores them in an inverted index (referred to hereinafter as a“similarity index”). In some embodiments, ‘m’=456 and ‘n’=64. In variousembodiments, the similarity index is stored in a logical partitionedspace within the database system 150. In alternate embodiments, thesimilarity index is stored in another database system 150′ co-locatedand in data communication with the database system 150 or, alternately,located remotely from the database system 150.

Thus, using the trained models 400 a, 400 b the content search module140 enables personal visual symbol search to identify offensiveplayer-generated personal visual symbol data or content (and quicklyenforce content policies and guidelines) that may not have shown up inthe top N results upon querying the database system 150, but arevisually similar to some known example. For example, the models 400 a,400 b may be used to search for certain types of personal visual symbols(for example, find all swastikas, and find all foul language) or reversesearch based on a personal visual symbol (find all personal visualsymbols similar to an offensive one). In some embodiments, the models400 a, 400 b query and search player-generated personal visual symboldata, similar to the input 2D/3D content, using a metric such as L2(Euclidean) distance or cosine angle.

FIG. 5 illustrates a workflow 500 implemented on the system 100 of FIG.1A for searching visual assets, including filtered or offensive content,in accordance with some embodiments of the present specification.

Referring to FIGS. 1A and 5, a user (such as a system administrator, forexample) uses the administrative work-station 145 to access the server105 and initiate a content search via content search module 140. Inembodiments, the content search module 140 presents the user with anasset search GUI 525 over the network 115. In embodiments, the userinteracts with the GUI 525 to initiate a search for assets similar toone or more 2D and/or 3D target personal visual symbols.

At step 516, the user, via GUI 525, inputs search criteria, in the formof image data, rendering instructions, textual descriptions, keywords,or other data, and the inputted search criteria is transmitted, by thecontent search module 140, to the trained models 400 a′, 400 b′ thatgenerate corresponding target asset signature(s). Preferably, the inputsearch criteria is in the form of at least one of image data orrendering instructions. In one embodiment, the input search criteria isin the form of image data and the content search module 140 translatesthe image data into a plurality of rendering instructions, inalphanumeric form, that is then inputted into the trained models 400 a′,400 b′

At steps 512, 518, the trained models 400 a′, 400 b′ generate, inresponse to the search criteria, a plurality of asset signaturesrepresentative of the inputted search criteria and queries an assetdatabase and/or similarity index to determine if the queried of assetsignatures are already stored and retrieve images or personal visualsymbols associated with, or embodying, the queried asset signatures.More specifically, a database 150 of asset signatures is queried withthe signatures determined from the inputted search query. Identifiedasset signatures that correspond to the search query are then inputted514 into a similarity index to find all images or personal visualsymbols that embody, or would be considered visually similar to, theidentified asset signatures. The identified images or personal visualsymbols are then communicated 520 back to the content search module 140and GUI 525 for viewing by the user. In some embodiments, the searchesare performed using a metric such as L2 (Euclidean) distance or cosineangle. In some embodiments, the database system 150′ is co-located andin data communication with the database system 150 or, alternately,located remotely from the database system 150. Alternately, in someembodiments, the similarity index is stored in a logical partitionedspace within the database system 150.

The plurality of asset signatures, stored in the asset database 150, aregenerated from a training system that acquires 2D texture assets 504 and3D model assets 506. In one embodiment, the 2D texture assets 504 and 3Dmodel assets 506 are maintained in logically separated data structures.In some embodiments, the assets 504, 506 are human-indexed or labeledfor supervised training. In some embodiments, the texture and modelassets 504, 506 are respectively 2D and 3D personal visual symbol data.At step 508, the training module 135 implements supervised training 508of the first and second machine learning models 400 a, 400 b (ML module126) using the assets 504, 506, respectively. Output of the supervisedtraining constitutes the trained models 400 a′, 400 b′ that, at step510, are deployed for use within the system 100, as discussed above.

The above examples are merely illustrative of the many applications ofthe system and method of present specification. Although only a fewembodiments of the present specification have been described herein, itshould be understood that the present specification might be embodied inmany other specific forms without departing from the spirit or scope ofthe specification. Therefore, the present examples and embodiments areto be considered as illustrative and not restrictive, and thespecification may be modified within the scope of the appended claims.

We claim:
 1. A method for generating and filtering digital media in amulti-player gaming network, wherein the multi-player gaming networkcomprises at least one game server and a plurality of client devices indata communication and located remote from each other, the methodcomprising: executing, in a game module stored locally in each of theplurality of client devices, a content editor application, wherein thecontent editor application is configured to generate a user interfacethrough which a player may create a personal visual symbol and isconfigured to generate personal visual symbol data based upon thepersonal visual symbol; receiving, in the at least one game server, theplayer-created personal visual symbol data from the game module;processing, in the at least one game server and using a contentclassification module, the player-created personal visual symbol data bysubmitting the player-created personal visual symbol data to a trainedclassification module; assigning, in the at least one game server andusing the content classification module, a value to the player-createdpersonal visual symbol data wherein the value is indicative of whetherthe player-created personal visual symbol data is or is not permissiblein the multi-player gaming network; and applying an action to theplayer-created personal visual symbol based upon said value, wherein theaction includes at least one of permitting the player-created personalvisual symbol to be used in the multi-player gaming network orprohibiting the player-created personal visual symbol from being used inthe multi-player gaming network.
 2. The method of claim 1, wherein thecontent classification module is configured to augment theplayer-created personal visual symbol prior to processing by the trainedclassification module.
 3. The method of claim 1, wherein the personalvisual symbol data comprises at least one of an image file or aplurality of rendering instructions in an alphanumeric format.
 4. Themethod of claim 1, further comprising generating multiple personalvisual symbols, in at least one of the plurality of client devices,wherein at least some of the multiple personal visual symbols compriseimagery designed to be not permissible in the multi-player gamingnetwork and at least some of the multiple personal visual symbolscomprise imagery designed to be permissible in the multi-player gamingnetwork.
 5. The method of claim 4, further comprising receiving thepersonal visual symbols which comprise imagery designed to not bepermissible in the multi-player gaming network and the personal visualsymbols which comprise imagery designed to be permissible in themulti-player gaming network.
 6. The method of claim 5, furthercomprising assigning one or more labels to each of the personal visualsymbols which comprise imagery designed to not be permissible in themulti-player gaming network and the personal visual symbols whichcomprise imagery designed to be permissible in the multi-player gamingnetwork, wherein each of the one or more labels comprises a valueindicative of whether a personal visual symbol is or is not to bepermitted in the multi-player gaming network.
 7. The method of claim 6,further comprising submitting each of the labelled personal visualsymbols which comprise imagery designed to not be permissible in themulti-player gaming network and the labelled personal visual symbolswhich comprise imagery designed to be permissible in the multi-playergaming network to at least one machine learning module, wherein the atleast one machine learning module is configured to generate the trainedclassification module.
 8. The method of claim 7, wherein at least one ofthe imagery designed to not be permissible in the multi-player gamingnetwork or the imagery designed to be permissible in the multi-playergaming network is submitted to the at least one machine learning modulein a form of alphanumeric text without an accompanying graphical image.9. The method of claim 1, wherein the multi-player gaming networkautomatically applies the action to the player-created personal visualsymbol based upon the value without human intervention.
 10. A system forgenerating and filtering digital media in a multi-player gaming network,wherein the multi-player gaming network comprises at least one gameserver and a plurality of client devices in data communication andlocated remote from each other, the system comprising: one or moreprocessors in a computing device, said one or more processors configuredto execute a plurality of executable programmatic instructions togenerate and filter digital media in the multi-player gaming network; agame module stored locally in each of the plurality of client devicesand configured to execute a content editor application, wherein thecontent editor application is configured to generate a user interfacethrough which a player may create a personal visual symbol and isconfigured to generate personal visual symbol data based upon thepersonal visual symbol; and a content classification module in the atleast one game server, configured to receive and process theplayer-created personal visual symbol data by submitting theplayer-created personal visual symbol data to a trained classificationmodule and to assign a value to the player-created personal visualsymbol data, wherein the value is indicative of whether theplayer-created personal visual symbol data is or is not permissible inthe multi-player gaming network, and wherein the content classificationmodule is configured to apply an action to the player-created personalvisual symbol based upon said value, wherein the action includes atleast one of permitting the player-created personal visual symbol to beused in the multi-player gaming network or prohibiting theplayer-created personal visual symbol from being used in themulti-player gaming network.
 11. The system of claim 10, wherein thecontent classification module is configured to augment theplayer-created personal visual symbol prior to processing by the trainedclassification module.
 12. The system of claim 10, wherein the personalvisual symbol data comprises at least one of an image file or aplurality of rendering instructions in an alphanumeric format.
 13. Thesystem of claim 10, wherein the content editor application is configuredto generate multiple personal visual symbols, in at least one of theplurality of client devices, wherein at least some of the multiplepersonal visual symbols comprise imagery designed to be not permissiblein the multi-player gaming network and at least some of the multiplepersonal visual symbols comprise imagery designed to be permissible inthe multi-player gaming network.
 14. The system of claim 13, wherein theat least one game server is configured to receive the personal visualsymbols which comprise imagery designed to not be permissible in themulti-player gaming network and the personal visual symbols whichcomprise imagery designed to be permissible in the multi-player gamingnetwork.
 15. The system of claim 14, wherein the content classificationmodule is configured to assign one or more labels to each of thepersonal visual symbols which comprise imagery designed to not bepermissible in the multi-player gaming network and the personal visualsymbols which comprise imagery designed to be permissible in themulti-player gaming network, and wherein each of the one or more labelscomprises a value indicative of whether a personal visual symbol is oris not to be permitted in the multi-player gaming network.
 16. Thesystem of claim 15, wherein the content classification module isconfigured to submit each of the labelled personal visual symbols whichcomprise imagery designed to not be permissible in the multi-playergaming network and the labelled personal visual symbols which compriseimagery designed to be permissible in the multi-player gaming network toat least one machine learning module, wherein the at least one machinelearning module is configured to generate the trained classificationmodule.
 17. The system of claim 16, wherein at least one of the imagerydesigned to not be permissible in the multi-player gaming network or theimagery designed to be permissible in the multi-player gaming network issubmitted to the at least one machine learning module in a form ofalphanumeric text without an accompanying graphical image.
 18. Thesystem of claim 10, wherein the content classification moduleautomatically applies the action to the player-created personal visualsymbol based upon the value without human intervention.
 19. A computerreadable non-transitory medium comprising a plurality of executableprogrammatic instructions wherein, when said plurality of executableprogrammatic instructions are executed by a processor in a computingdevice, a process for generating and filtering digital media in amulti-player gaming network is performed, wherein the multi-playergaming network comprises at least one game server and a plurality ofclient devices in data communication and located remote from each other,the plurality of executable programmatic instructions comprising:programmatic instructions, stored in the computer readablenon-transitory medium, for generating and filtering digital media in amulti-player gaming network by: executing, in a game module storedlocally in each of the plurality of client devices, a content editorapplication, wherein the content editor application is configured togenerate a user interface through which a player may create a personalvisual symbol and is configured to generate personal visual symbol databased upon the personal visual symbol; receiving, in the at least onegame server, the player-created personal visual symbol data from thegame module; processing, in the at least one game server and using acontent classification module, the player-created personal visual symboldata by submitting the player-created personal visual symbol data to atrained classification module; assigning, in the at least one gameserver and using the content classification module, a value to theplayer-created personal visual symbol data wherein the value isindicative of whether the player-created personal visual symbol data isor is not permissible in the multi-player gaming network; and applyingan action to the player-created personal visual symbol based upon saidvalue, wherein the action includes at least one of permitting theplayer-created personal visual symbol to be used in the multi-playergaming network or prohibiting the player-created personal visual symbolfrom being used in the multi-player gaming network.
 20. The computerreadable non-transitory medium of claim 19, wherein the contentclassification module is configured to augment the player-createdpersonal visual symbol prior to processing by the trained classificationmodule.
 21. The computer readable non-transitory medium of claim 19,wherein the personal visual symbol data comprises a plurality ofrendering instructions in an alphanumeric format representative of animage and does not include an image file.
 22. The computer readablenon-transitory medium of claim 19, wherein the multi-player gamingnetwork automatically applies the action to the player-created personalvisual symbol based upon the value without human intervention.